This application incorporates by reference in its entirety a computer program listing appendix containing the source code-listings which were described in the patent application at the time of filing and which have now been reproduced on compact disk. The compact disk contains the file entitled xe2x80x9cAPPENDIX (SOURCE CODE),xe2x80x9d was created on Jun. 18, 2003, and encompasses 812 kilobytes.
1. Field of the Invention
The present invention relates to interactive computer systems, and in particular to a method and apparatus for facilitating wireless, full-body interaction between a human participant and a computer-generated graphical environment including textual data.
2. Description of Related Art
So-called xe2x80x9cvirtual realityxe2x80x9d (xe2x80x9cVRxe2x80x9d) systems enable users to experience computer-generated environments instead of merely interacting with them over a display screen. Such systems typically require the user to don goggles, through which he or she perceives the virtual environment, as well as sensors that encode the user""s gestures as electrical signals. The user reacts naturally to the changing virtual environment, generating signals that the computer interprets to determine the state and progress of the presented environment.
In order to encode a sufficiently broad spectrum of gestures to facilitate natural interaction, VR systems ordinarily require the user to wear, in addition to the goggles, at least one xe2x80x9cdata glovexe2x80x9d to detect hand and finger movements, and possibly a helmet to detect head movements. Full-body systems, which encode movements from numerous anatomical sites to develop a complete computational representation of the user""s overall body action, require many more sensors; however, such systems, would be capable of projecting the user fully into the virtual environment, providing the user with greater control and a heightened sense of participation ideally suited to interactive simulations.
Unfortunately, numerous practical difficulties limit the capacity of current VR systems to achieve this goal. The nature of the interaction currently offered, even with full-body sensor arrays, is limited. The computational demands placed on a system receiving signals from many sensors can easily overwhelm even large computers, resulting in erratic xe2x80x9cjumpsxe2x80x9d in the visual presentation that reflect processing delays. Moreover, no matter how many sensors surround the user, they cannot xe2x80x9cseexe2x80x9d the user, and therefore cannot integrate the user""s true visual image into the virtual environment.
Economic and convenience factors also limit sensor-type VR systems. As the capabilities of VR systems increase, so do the cost, awkwardness and inconvenience of the sensor array. The sensors add weight and heft, impeding the very motions they are intended to detect. They must also ordinarily be connected, by means of wires, directly to the computer, further limiting the user""s movement and complicating equipment arrangements.
In order to overcome the limitations associated with sensor-based VR systems, researchers have devised techniques to introduce the user""s recorded image into a virtual environment. The resulting composite image is projected in a manner so that it may be viewed by the user, enabling the user to observe his or her appearance in and interaction with the virtual environment.
Three such approaches include the VideoPlace system (see, e.g., M. Krueger, Artificial Reality II (1991) and U.S. Pat. No. 4,843,568), the Mandala system (see, e.g., Mandala VR News, Fall/Winter 1993; Vincent, xe2x80x9cMandala: Virtual Villagexe2x80x9d and Stanfel, xe2x80x9cMandala: Virtual Cities,xe2x80x9d Proceedings of ACM SIGGRAPH 1993 at 207-208 (1993)), and MIT""s system disclosed in U.S. Pat. No. 5,563,988. Unfortunately, these systems exhibit various limitations. For example, Krueger""s VideoPlace requires a special background and ultraviolet lamps, and extracts and represents only the user""s silhouette. The Mandala system can integrate the user""s full image within the virtual environment it creates, but requires a chroma-key blue background. The system developed by MIT provides a 3-Dimensional spatial video interaction with a user and is directed to detecting and analyzing gestural information of the user to cause predetermined responses from graphically generated objects within the virtual environment (e.g., if a virtual user pets a graphically generated dog within the virtual environment, the dog may wag its tail). Regardless of the systems, all of the above-described VR systems employ complex video analysis and tracking techniques to detect and follow the boundaries of the user""s image within the virtual environment.
The disclosure of U.S. Pat. Nos. 5,563,988 of Maes et al, and 4,843,568 of Krueger et al. are hereby incorporated by reference as if set forth in their entirety herein.
Applicant has recognized the need for a simple 2-dimensional video-generated real-time interaction system wherein text data is easily introduced into a virtual environment, as viewed on a monitor or on a projection screen. The inputted text data is manipulated in the virtual environment by video representation of a human participant (a xe2x80x9cvirtual userxe2x80x9d). Through movements in the real world, the human participant or virtual user freely interacts with the inputted text data in the virtual environment as the human participant (and other people watching the monitor or projection screen) absorbs the teachings of the text data as he or she interacts with the text data in the virtual environment.
It is an object of the present invention to provide a wireless VR system that easily accepts text data from a text input to be displayed onto a monitor (or projection screen) so that the font, size, shape, and/or movements of words and/or characters (e.g, letters, symbols and numbers) of the inputted text data may be altered by a virtual user (a real-time video representation of a human participant).
It is another object of the invention to enable the virtual user to interact with computer-generated, visually represented text data in such a manner that allows the inputted text to be readable within the virtual environment so that the virtual user may learn, understand, or otherwise experience the meaning of the textual data as he or she interacts with the text within the virtual environment.
It is another object of the invention to provide an interaction display system wherein text data and the video image of a user is combined within a virtual environment to promote marketing, advertisement, sales, and entertainment.
It is another object of the invention to provide an interaction display system wherein text data and the video image of a user is combined within a virtual environment and wherein the text data viewed within the virtual environment may be changed in realtime in response to actions by the user or by others.
It is another object of the invention to provide an interaction display system wherein text data and the video image of a user is combined within a virtual environment and wherein the text data is imported from a local text file.
It is another object of the invention to provide an interaction display system wherein text data and the video image of a user is combined within a virtual environment and wherein the text data is imported from a remote site over the Internet.
It is another object of the invention to provide an interaction display system wherein text data and the video image of a user is combined within a virtual environment and wherein the text data is inputted directed from a keyboard.
The foregoing and other objects of this invention can be appreciated from the Summary of the Invention, the drawing Figures and Detailed Description.
In accordance with the invention, a controlling software program instructs an interactive display system. The interactive display system includes an electronic camera (e.g., charge-coupled device or xe2x80x9cCCDxe2x80x9d) which intermittently records (in realtime) a human participant""s (a user) image and inputs the video image into a computer. The computer creates a digitized version of the inputted image (e.g., a bitmap of pixels) and thereby creates a xe2x80x9cvirtual user imagexe2x80x9d. According to one embodiment, the camera is positioned behind a large viewing screen which faces the user and records the user""s image through an opening provided within the screen. The computer further receives a data record including text data and text controlling parameters, preferably including at least a fall rate value. The data record may be supplied to the computer using any of several conventional memory mediums, such as a hard drive memory, disc memory, compact disc, or the data record may only include a text string which is supplied to the computer directly from the Internet through a connected modem, for example, or through the use of a connected keyboard. Alternately, the text may be inputted to the computer directly from a word processing program, such as Corel""s WordPerfect(copyright), or Microsoft Word(copyright) as a text file or by xe2x80x9ccopyingxe2x80x9d selected text from an open text file directly into the present software program.
According to the invention, upon receiving a captured video image of the user, the software program (see appendix for source code) first parses the text data into lines of text. The program determines which line or lines of text should be displayed on the screen and further determines the destination of each character within any line that is to be displayed on the screen, according to the fall rate assigned to each character. The program then compares the pixel color value of xe2x80x9cdestination pixelsxe2x80x9d (pixels located at the destination of each particular character) with a threshold color value. If the color value of the destination pixels is below the threshold value (i.e., lighter), then the particular character will be positioned at a destination according to its prescribed fall rate. If, however, the color value of the destination pixels exceeds the threshold value (i.e., darker), then it is determined that the virtual user""s image is located at the destination of the particular character of the line of text. In such instance, the program locates a suitable new destination positioned vertically above the original destination which is lighter in color value than the threshold value.
Once all the characters of all the lines of text intended to be projected onto the screen are provided with suitable destinations, the program instructs the computer to display all of the characters on the screen at their respective destinations with the particular captured video frame. The process continues when a new image frame is available from the camera.
The program analyzes each captured image as it acquires at a prescribed sampling rate. The sampling rate is dependent on the speed of the video card components and drivers, the speed of the computer, and light sensitivity of the digital camera.
According to the invention, as the individual characters of a single line make their way down along the projection screen, the program keeps track of when a predetermined number of characters (e.g., at least half of them) travels a predetermined distance from the top of the viewing screen (such as the halfway point). If this is the case, the program causes the entire line of text to fade at a prescribed fade rate. When one line fades at the lower portion of the screen, another line appears at the top of the screen, with its characters beginning to fall. It is preferred that at least three lines of text be displayed on the screen at any given time.
Another aspect of the invention displays each line of text data as a separate color so that viewers may easily discern the letters or characters from different lines of inputted text, allowing the lines of text to be read relatively easily, as the letters fall towards and rest on various darker elements located within the virtual environment, such as the arm of a virtual user and perhaps a xe2x80x9cvirtual umbrella imagexe2x80x9d carried by the virtual user.
According to another aspect of the invention, each character of an inputted text string is horizontally held so that each character is restricted to vertically movement as it descends on the viewing screen at its prescribed fall rate or ascends upwardly with similar upward movement by the virtual user.
According to another embodiment of the invention, the program generates a shadow zone of dark pixels which are projected onto the viewing screen. The shadow zone is aligned with the optical input of the digital camera so that the camera will be protected from the relatively bright light of the projector. The computer generated shadow zone helps prevent the images recorded by the camera from becoming xe2x80x9cwashed outxe2x80x9d by the light from the projector shining on the camera""s lens.